Move plumber2 scripts to python directory #2503

TeaganKing · 2024-05-01T16:28:11Z

Description of changes

This PR addresses @2187 by moving new scripts that were added in #2155 to tools/site_and_regional into the python directory, similarly to other NEON scripts in #2156.

Fixed: #2187

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? No

Testing performed, if any:
I tested newly added unit test, which was successful, with python -m unittest test_unit_plumber2_surf_wrapper.py. I will add that when testing with ./run_ctsm_py_tests -u, I ran into a seemingly unrelated error that I'll add in a comment below.

These failures are already tracked with derecho versions

This will be reverted once issue ESCOMP#2423 has been addressed

This also adds a corresponding namelist build unit test and similar test to check that sp mode + spitfire fails

Merge b4b-dev

Merge b4b-dev Some fixes for run_neon that came up in ctsm5.1.dev172. Small bug fix for units of vector format hillslope hydrology. Start adding PLUMBER2 users-mods (NOT functional)

FATES API35 parameter file update

TeaganKing · 2024-05-01T16:29:09Z

Here is the probably unrelated error in testing that I mentioned above:

./run_ctsm_py_tests -u
...........................................E
Stdout:
ERROR while running:
/glade/work/tking/ctsm_2187/CTSM/cime/CIME/scripts/configure --macros-format CMake --silent --compiler intel --machine derecho
From /glade/derecho/scratch/tking/tmp/tmp86u2eboo/tools_bld

ERROR: module command /glade/u/apps/casper/23.10/spack/opt/spack/lmod/8.7.24/gcc/7.5.0/m4jx/lmod/lmod/libexec/lmod python load craype intel/2023.2.1 mkl ncarcompilers/1.0.0 cmake cray-mpich/8.1.27 netcdf-mpi/4.9.2 parallel-netcdf/1.12.3 parallelio/2.6.2 esmf/8.6.0 failed with message:
Lmod has detected the following error: The following module(s) are unknown: "craype"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "craype"

Also make sure that all modulefiles written in TCL start with the string #%Module


...............................................................................................................................................................
======================================================================
ERROR: test_derecho_mpirun (test.test_unit_gen_mksurfdata_jobscript_single.TestFGenMkSurfJobscriptSingle)
test derecho mpirun. This would've helped caught a problem we ran into
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/work/tking/ctsm_2187/CTSM/python/ctsm/os_utils.py", line 21, in run_cmd_output_on_error
    _ = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True, cwd=cwd)
  File "/glade/u/home/tking/.conda/envs/ctsm_pylib/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/glade/u/home/tking/.conda/envs/ctsm_pylib/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/glade/work/tking/ctsm_2187/CTSM/cime/CIME/scripts/configure', '--macros-format', 'CMake', '--silent', '--compiler', 'intel', '--machine', 'derecho']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/glade/work/tking/ctsm_2187/CTSM/python/ctsm/test/test_unit_gen_mksurfdata_jobscript_single.py", line 161, in test_derecho_mpirun
    cmd=cmd_list, errmsg="Trouble running configure", cwd=self._bld_path
  File "/glade/work/tking/ctsm_2187/CTSM/python/ctsm/os_utils.py", line 30, in run_cmd_output_on_error
    abort(errmsg)
  File "/glade/work/tking/ctsm_2187/CTSM/python/ctsm/utils.py", line 26, in abort
    sys.exit("ERROR: {}".format(errmsg))
SystemExit: ERROR: Trouble running configure

Stdout:
ERROR while running:
/glade/work/tking/ctsm_2187/CTSM/cime/CIME/scripts/configure --macros-format CMake --silent --compiler intel --machine derecho
From /glade/derecho/scratch/tking/tmp/tmp86u2eboo/tools_bld

ERROR: module command /glade/u/apps/casper/23.10/spack/opt/spack/lmod/8.7.24/gcc/7.5.0/m4jx/lmod/lmod/libexec/lmod python load craype intel/2023.2.1 mkl ncarcompilers/1.0.0 cmake cray-mpich/8.1.27 netcdf-mpi/4.9.2 parallel-netcdf/1.12.3 parallelio/2.6.2 esmf/8.6.0 failed with message:
Lmod has detected the following error: The following module(s) are unknown: "craype"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "craype"

Also make sure that all modulefiles written in TCL start with the string #%Module



----------------------------------------------------------------------

ekluzek · 2024-05-01T16:39:05Z

@TeaganKing thanks for working on bringing this in.

A small formatting suggestion for PR's and issues. When you have test results that come in, it'll format it weirdly naturally. But, if you highlight and make it code with the "<>" option above the comment it'll look more normal.

TeaganKing · 2024-05-01T16:49:08Z

Thanks @ekluzek . I had pressed '<>' before adding the code, and it's certainly not a one-line bit of code, so the single quotes didn't work as I intended-- I fixed that though :)

I think I also must have not set up merging into b4b dev correctly, as this is also pulling in changes from you and Gregory? I'm assuming these changes should go into b4b dev before master?

ekluzek · 2024-05-01T16:53:00Z

@TeaganKing I'm thinking you just need to merge in the latest escomp/b4b-dev. Fetch and then do that and it should clean that up. If not we'll need to figure that out.

One other suggestion is that you make sure the issues fixed are on it's own line with a "fixed" in front of it. That will it will know to autoclose the issue. This is something we should make more obvious in the PR template.

TeaganKing · 2024-05-01T17:34:24Z

This is now being done in #2505 so that it is off of the correct branch... closing this PR.

glemieux and others added 15 commits March 25, 2024 09:54

update default fates parameter file

c316b7f

update fates externals

618a083

remove out-of-date cheyenne expected failures

5c98773

These failures are already tracked with derecho versions

add failing 5x5_amazon test to aux_clm and known failures to track

7481a5a

temporarily convert fates seed dispersal test to use f10

e9fb075

This will be reverted once issue ESCOMP#2423 has been addressed

add graceful failure for fates hydro + sp mode

3c73756

This also adds a corresponding namelist build unit test and similar test to check that sp mode + spitfire fails

Merge tag 'ctsm5.1.dev176' into fates-api35-param_update

3bd358d

Merge b4b-dev

Merge tag 'ctsm5.2.001' into fates-api35-param_update

12dd7c0

Merge b4b-dev Some fixes for run_neon that came up in ctsm5.1.dev172. Small bug fix for units of vector format hillslope hydrology. Start adding PLUMBER2 users-mods (NOT functional)

remove fates sp mode expected fail per sci.1.72.7_api.34.0.0 tag fix

810272b

update compset for 5x5_amazon failing test

3425304

Update ChangeLog and ChangeSum

9b1355f

A few changes based on my review as author is out

f72e9cf

Update date

e0e6f11

Merge pull request ESCOMP#2436 from glemieux/fates-api35-param_update

286146f

FATES API35 parameter file update

move plumber2 scripts to python directory and add testing

40bf2e5

TeaganKing self-assigned this May 1, 2024

fixed all pylint errors in plumber2_*.py files and ran black

ff2aa51

TeaganKing linked an issue May 1, 2024 that may be closed by this pull request

Move new PLUMBER2 scripts to python directory to enable python testing #2187

Closed

TeaganKing closed this May 1, 2024

TeaganKing mentioned this pull request May 1, 2024

Move plumber2 scripts to python directory (on b4b branch) #2505

Merged

TeaganKing deleted the move_scripts_plumber branch June 24, 2024 19:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move plumber2 scripts to python directory #2503

Move plumber2 scripts to python directory #2503

TeaganKing commented May 1, 2024 •

edited

Loading

TeaganKing commented May 1, 2024 •

edited

Loading

ekluzek commented May 1, 2024

TeaganKing commented May 1, 2024 •

edited

Loading

ekluzek commented May 1, 2024

TeaganKing commented May 1, 2024

Move plumber2 scripts to python directory #2503

Move plumber2 scripts to python directory #2503

Conversation

TeaganKing commented May 1, 2024 • edited Loading

Description of changes

TeaganKing commented May 1, 2024 • edited Loading

ekluzek commented May 1, 2024

TeaganKing commented May 1, 2024 • edited Loading

ekluzek commented May 1, 2024

TeaganKing commented May 1, 2024

TeaganKing commented May 1, 2024 •

edited

Loading

TeaganKing commented May 1, 2024 •

edited

Loading

TeaganKing commented May 1, 2024 •

edited

Loading